Repository navigation
Warn about dates that may be real in the privacy check - #187
Merged
Merged
Conversation
The README and check_privacy docstring said dates were checked, but no date check existed. Add one: dates between date_lookback_years ago (200 by default) and today are reported as possibly real, since de-identified datasets usually shift dates. Future dates are treated as shifted. Written dates are found in CSV cells and text files. Unix seconds and milliseconds are only read in columns with time-like names, or in layout timestamp columns and metadata rows such as E4 start times, because IDs and counters have the same shape. Each file gets one warning listing its columns and date ranges; after 10 files the rest are summarized. E4 test fixtures now use timestamps shifted to 2150, so format tests stay independent of the date check.
…ding
A day- or month-first date followed by a full stop ("13/01/2023.") was
missed. A trailing period is now allowed; a decimal extension still is not.
date_lookback_years must be a whole number from 1 to 1000 or None.
Negative values silently matched nothing, 0 turned the check off, and very
large values raised an error partway through validation.
Every file with possible real dates now keeps its own warning, tagged with
the code "possible-real-date", so JSON and Markdown reports list them all.
Only the terminal output is shortened, to 10 warnings of one kind per
category plus a count of the rest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The README and
check_privacydocstring said the privacy check looked for date patterns, but no date check existed. This adds one..txtfiles: year-first (2023-01-13, with optional time), day- or month-first with a four-digit year (13/01/2023), and month names (13 Jan 2023,January 13, 2023,Jan 2023). Sentence-ending dates are included.charttime,timestamp,ts,event_time), in a layout's timestamp columns, and in metadata rows such as Empatica E4 start times. IDs and counters have the same shape, so other numeric columns are not read as timestamps.20230113), times without a date, and years on their own.possible-real-date. The terminal shows the first 10 per category and counts the rest.ValidationConfig.date_lookback_years(default 200). It accepts a whole number from 1 to 1000, orNoneto turn the check off; other values raiseValueError.start_time,sample_rate) so warnings point at the start time.Notes
LICENSE.txtcontains a 1996 date..csv.gz) are not scanned by any check; that is unchanged here.Verification
pytest -q: 257 passed, 1 skippedgit diff --check: passed